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ABSTRACT 

This study developed a classification scheme to 
critically compare performance assessment projects at higher 
education universities in North America and Europe. Performance 
indicators and assessment initiatives were compared using nine basic 
dimensions: (1) locus of control, (2) degree of governmental 
involvement, (3) focus of performance indicators, (4) sources of 
quality variation, (5) data selection, (6) intended audiences, (7) 
emphasis of use, (8) impact on student learning, and (9) relationship 
to institutional mission. A sample of six systems of generation 
performance indicators was examined using the nine criteria; the six 
systems are: National Education Goals (United States National 
System); Critical Choices (United States State System); Key Success 
Indices (United States Institutional System); The Committee for Vice 
Chancellors and Principals listing of performance indicators (British 
National System); Queen's University (Canadian Model); and MONEY 
magazine (United States Media System). Analysis found that 
performance indicators are mostly variations of input/output 
mechanistic thinking, that remarkable similarities exist between 
North American and European initiatives, that most models are 
unprepared to address conversion or process variables, that most are 
built on an assumption that outcomes can be a^^.ributed to something 
in the system or institution, and that the absence of linkages back 
to the learning environment leaves internal decisionmakers without 
information to correct causes that explain variations in quality. 
(Contains 92 references.) (JB) 
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PERFORMANCE INDICATORS AND RATIONAL MANAGEMENT TOOLS; 
A COMPARATIVE ASSESSMENT OF PROJECTS 
IN EUROPE AND NORTH AMERICA 



higher education at public and independent universities in North America and Europe. 
This paper represents work in progress that develops a classificatory schema to 
critically compare these rational management initiatives along several dimensions, e.g., 
conceptual underpinnings, governance, levels of aggregation, usage patterns, and the 
like. The paper concludes with a discussion of the implications of these initiatives for 
organizational change and some lessons for institutional researchers in North America 
and Europe. 

INTRODUCTION 

Higher education in Europe and North America is under review from a variety 
of quarters to document the quality of effort in achieving educational goals. The 
notions of accountability and quality are at the forefront of discussions today because 
the basic metaphor guiding the dialogue about education has changed. Conceptually, 
higher education has moved from a type of "public utility" to a "strategic investment" 
(Ewell, 1991). Effective performance has become a part of the contemporary lexicon 
about excellence and quality as the public debates society's return on its investment. 

This paper describes the current statu of affairs in the design and 
implementation of assessment mechanisms in a variety of settings in North America 
and Europe. Second, these performance indicator and assessment initiatives are 
compared in terms of nine basic dimensions including (1) locus of control, (2) degree 
of governmental involvement, (3) focus of performance indicators, (4) sources of 
quality variation, (5) data selection, (6) intended audiences, (7) emphasis of use, (8) 
impact on student learning, and (9) relationship to institutional mission. The paper 
closes with a discussion of implications for institutional researchers, and for the higher 
education communities in North America and Europe. 
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OVERVIEW 

Critics have begun to examine meanings about excellence and quality that have 
been developed by stakeholders in higher education. For example, assessment projects 
are beginning to move beyond traditional reputational and resource approaches to a 
"talent development" perspective (Astin, 1985; Jacobi, Astin, and Ayala, 1987). This 
changed emphasis is part of a larger movement to gradually introduce a rational 
management perspective in the operation and governance of higher education. An 
emphasis on strategic investment has v reated its own architecture that, in turn, has 
shaped the design and implementation of performance assessment systems. Other 
rational management tools under the broad umbrella of quality assurance carry such 
labels as student achievement/outcome asse-^sment, faculty productivity measurement, 
and administrative performance review. 

Whether symbolic or substantive, performance indicators as a class of rational 
management quality assurance tools have been introduced by a host of agencies. State, 
federal or national governments in many countries are calling for assessment systems 
through executive initiatives or accountability legislation. Regional or national 
administrative agencies, such as accreditation review activities, have presented a ^de 
range of approaches to accountability measurement. 

More recently, mass media publications in Europe and North America have 
discovered a political vacuum in the higher education environment and have created 
numerous "reports" that mascarade as measures of quality. For example. Money 
magazine, IIS News and World Report . Barron's Pro files of American Colleges. 
Peterson's /ACiB Survey of Strategic Indicators . Mafilfianis, and the "league tables" in 
the United Kingdom are evidence of the growing popularity of public rankings. From 
some perspectives, this unfortunate movement toward undisciplined consumerism 
appears to be gaining momentum. Although this approach has been labeled 
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"fundamentally dangerous" and "largely devoid of meaning," (Webster, 1992) their 
popularity is growing and their effect on institutional decision making is increasing, 

A rich variety of performance assessment initiatives has been launched in the 
past twenty years. A sample of the range of programs in Europe include: OECD's 
International Education Indicators project (Bottani and Delfau, 1990); Great Britain's 
Management Statistics and Performance Indicators (Jarratt Report, 1985; Committee 
of Vice-Chancellors Principals/Universities Funding Council, 1990; Ball and 
Halwachi, 1987; Gordon, 1992), and projects in Holland, Germany, Finland, 
Netherlands, and Sweden (Acherman, H. van Welie, L. and Laan, C, 1992; Maassen 
and Van Vught, 1988; Frackmann, 1987; Holtta, 1988; Mertens and Bormans, 1990; 
PoUitt, 1989; and among others, Kells, 1990). 

Projects in the United States from state and federal developments are 
illustrated by the California State Uiiiversity efforts (CSU Advisory Committee on 
Student Outcome Assessment, 1989), Missouri's Task Force on Critical Choices for 
Higher Education (Coordinating Beard for Higher Education, 1992), Tennessee's 
Performance Funding (Banta, 1986), Ohio's Selective Excellence Program (NCHEMS, 
1993), regional accrediting initiatives (North Central Association, 1993); and among 
others, the US Cor ;-*ssionally-mandated indicator system (National Center for 
Educational Statistics, 1991). Several Canadian initiatives are underway to develop 
and implement performance indicate systems (Organization for Economic Co- 
operation and Development, 1988; Snowdon, 1993; Association of Universities and 
Colleges of Canada, 1993; Smith, 1992). 

The U.S. government entered the debate on performance assessment with its 
much-publicized A Nation at Risk (US Department of Education, 1983), and continues 
to expand its intrusion into the affairs of higher education institutions. For example, 
the Student Right-to-Know and Campus Security Act (PL 101-542) responds to the 
growing consumer movement in education by requiring the posting of graduation rates 
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of a schoors first-time, full-time freshmen within a six-year period of entry. More 
recently, the reauthorization of the Higher Education Act (1992) in the 102nd 
Congress critically examined the role of accrediting bodies in determining the financial 
aid eligibility of institutions. Part H of the amended act, known as the Program 
Integrity Triad, introduces a variety of measures involving state postsecondary review 
of institutional financial aid and program integrity. 

Accrediting agencies and professional licensure bodies are additional forces 
that keep the quality assurance movement at tLe forefront of higher education. 
Regional accrediting bodies, however, have moved beyond their traditional view of 
performance in recent years. The North Central Association, for example, has shifted 
its focus from the institution's capacity to achieve its mission to accomplishments as 
expressed by student achievement. Simply stated, the accrediting days are gone when 
site visitation teams would be preoccupied with resources, e.g., counting books in the 
library or the number of Ph.D.'s on staff; the emerging questions for these assessors 
focus on what the students have done or achieved as a result of having experienced the 
institution. This changed emphasis is part of a larger movement to gradually introduce 
quality assurance assessment in the operation and governa"':^e of higher education. 

BASIC CONCEPTS 

Before attempting a comparison of performance assessment initiatives across 
two continents, this section will provide a clarification of vague and conflicting 
terminology related to this latest, and apparently, enduring movement in higher 
education. This section will examine the concepts of quality, assessment, institutional 
effectiveness, student outcomes, and process to inform the development of a series of 
dimensions for the systematic comparison of performance indicator systems. 
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QUALITY: Given the multiple stakeholders in higher education (faculty, students, staff, 
alumni, private and public sector employers) and their multiple visions of quality 
assurance, it is hardly surprising to find a wide variety of conceptualizations of quality- 
Some observers are led to the conclusion that "reaching consensus about exact 
definitions for quality is virtually impossible, and it is best to avoid having to do so" 
(Massey and Wilger, 1992, p. 364). Others have suggested that two cultures exist in the 
academy, with a corporate view that differs from the academic culture (Newton, 1992). 
When viewed as the mantra of management (e.g., the Total Quality Management or 
TQM perspective), quality is primarily concerned with exceeding the needs of the 
consumer (Deming, 1986). The "consumer" and quality are situationally-bounded, that 
is, during one transaction an individual may be a beneficiary of service, while in 
another they may be contributing to a service unit's realization of quality. The 
implication is that TQM defines quality in a way that ties indicators to specific 
contexts. (Chaffee anu Sherr, 1992). 

Much of the literature links quality assessment to basic design principles, e.g., 
organizational mission and goals, needs and program components. Thus, some 
students of continuous quality improvement are concerned with fidelity to design or 
what has been called "conformance to requirements" (Crosby, 1979). For Bogue and 
Saunders (1992, p. 20) quality is: 

Conformance to mission specification and goal achievement-within 
publicly accepted standards of accountability and mtegrity. 

This definition is particularly useful for two reasons. First, such a definition 
helps focus debate and shared thinking on the purposes of the enterprise, or what the 
institution intends to achieve. Second, such a conceptualization of quality encourages 
public disclosure of mission, goals, and performance results: 
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Quality assurance in higher education is a mosaiac, like the higher 
education enterprise itself. Mechanisms such as accreditation, 

Professional licensure, program review, and outcomes assessment each 
ave a distinctive part to play-and, if infused by a pervasive attitude of 
concern about students and society, they will sum to a condition that can 
rightly be labeled 'quality' (Ewell, 1992a). 

ASSESSMENT: Conformance to mission and goals is the substance of assessment. It 
may be referred to as an "honest match" between what higher education institutions say 
they will dc and what they deliver. For this study, assessment can be defined as 
follows: 

The process of measuring, by some appropriate means, the presence or 
absence of expected educational results. 

These expected educational results are what Ewell (1992b) refers to as the objects of 
assessment. Expected results flow directly from the mission and goals of the 
institution, and can be identified at three or more levels, such as the institution, 
programs, and individuals. 

Equally important, assessment involves methods by which evidence informs an 
observer about the extent to which conformance has occurred. Ewell (1992b) refers to 
this second side of assessment as the processes of assessment. To illustrate, suppose 
that one institutional goal was as follows: 

The University seeks to prepare individuals who can act as transforming 
agents in society. 

The expected educational results that flow from this goal statement might include a 
range of qualities, such as citizenship skills, basic problem-solving and communicative 
skills, and knowledge of a discipline or profession. Each of these goal areas are further 
refined into measurable indicators through the process of operationalization. 

This deductive process leading toward measurable indicators is a heavily value- 
laden activity. The normative character of the assessment process is understandable 
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not only because of the intimate connection between institutional goals and indicator 
construction, but also because of the effect of the public policy process: 

Govemmentally controlled indicators have tended to reflect each 
historical era's predominant political and social ideologies, and measures 
have been developed in response to (or occasionally in reaction against) 
prevailing political, social, and economic goals for schooling (Darline- 
Hammond, 1992, p. 237). eve 



Government involvement in assessment plans may lead to a centralization of 
control over indicator development, and discourage institutions from maintaining 
diversity or distinctivene.'is in their institutional mission. As Ball and Wilkinson (1992, 
p. 11) remarked about the British model: 

The danger of a national system of performance indicators such as those 
produced by the CVCP is that it will inevitably tend to nationalize 
institutional policy making. 

It is interesting to note that in England and Wales, efforts are underway to introduce a 
national assessment program along with a national curriculum for children at ages 7, 
11, 14, and 16 (Torrance, 1993). Whether or not this precollegiate effort will create a 
force on postsecondary educational practices in Great Britain remains to be seen. 

Another dimension of indicator development suggesting a normative; bias in the 
process is the tendency to embrace an item because it is easily measured (Elton, 1987; 
Bail and Wilkinson, 1992). That is, if data are available, they must be measuring 
something. Ease of quantification overcomes the need to have a conceptually 
defensible system of indicators. Simply stated, any method that is unguided by basic 
theory to assist the selection process lays the groundwork for rank empiricism of the 
highest order. The orientation becomes, "if it's measurable, it's worth something." 

The tendency to gravitate toward easily quantifiable measures, coupled with the 
potential tor centralizing measures (i.e., state or national policy making), may create a 
legislative climate that supports the "quick fix" approach. Because the policy process is 
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SO fluid and demands are cacophonous and continuous, the drive toward finding "the 
number" is highly seductive. When the legislative debate during appropriations 
hearings in legislatures focuses on the number of students graduating, the number of 
children served, the expenditures per FTE student, and the like, the quantitative 
expression of a preferred state of affairs is easily and readily translated into a 
budgetary target. 

INDICATORS: There exists a rich array of orientations toward indicator development 
and use. Indicators are "individual or composite statistics that reflect important 
features of a system, such as education, health, or the economy" (Darling-Hammond, 
1992, p. 236). Most indicator systems function to (a) monitor the broad context within 
which a policy is operative; (b) provide benchmark measurements against specified 
goals; (c) forecast the emergence or existence of new problems; and (d) permit the 
development of systematic explanations for the existence of problems. 

Regardless of their functional emphasis, postsecondary educational indicators 
continue to shape the policy debates and agendas of the governments of Europe and 
North America. In the United States, for example, much of the historical dialogue in 
postsecondary education centered on "access" indicators (proportion of minority 
enrolled), then later, the proportion graduating within a six-year period. As the 
indicator development process continued to grow, so did the sheer number of 
indicators. For example. Queen's University in Canada tracks 20 measures in four 
domains (Smith, 1992). The CVCP system in Great Britain includes fifty-four 
measures of the higher education system. The University of Miami's Key Success 
Indices (KSI) tracking system monitors 126 indices against prior month, year and, 
where appropriate, current budget levels (Sapp and Temares, 1992). 
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Performance indicators are a specialized subset of system indicators. They are 
used to assess the relative position of a system against some standard or reference 
point, such as an organizational goal (Cave, Hanney, and Hogan, 1992), While 
introduced to higher education far earlier in the United States than in Europe, several 
common characteristics exist regardless of the setting: (a) They tend to be expressed 
quantitatively, (b) encourage a comparative application through the use of rankings 
regardless of their methodological limitations, (c) are classified according to functions 
or stages (input, process, output), and most importantly, (d) are developed in the 
absence of a robust conceptual framework. In the absence of agreed upon statements 
of postsecondary goals, however, the development of comparative assessments is very 
difficult (Cuemin, 1987). 

Some have argued for "families of authorities," that is, institutions with shared 
characteristics, to form comparative sets of schools (Ball and Wilkinson, 1992). 
Queen's University in Canada recommends selecting comparable institutions on the 
basis of minimum PTE undergraduate and graduate enrollments and sponsored 
research threshholds (Smith, 1992). Others have warned that a national system of 
indicators undermines the distinctiveness of institutional missions and contexts (Ball 
and Wilkinson, 1992). 

Since performance indicators tend to be developed in relationship to goals, 
early writings distinguish between outcome and process goals. Outcome performance 
indicators are *'substantive objectives of the institution, [the latter] relate mainly to the 
internal performance of the organization" (Ball and Halwachi, 1987). We will take a 
closer look at process variables later on in that section on basic concepts. 

iNsnrunoNAL effectiveness: in the development of indicators as management 
tools, much of the design language seems to borrow from the industrial model, drawing 
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important distinctions among such terms as organizational effectiveness, efficiency, and 
economy (Sizer, 1989). Equally significant, the basic governance structure of higher 
education does not lend itself to the direct application of the industrial input/output 
model, with its dual system of control between administrative and professional cultures 
(De Jager, 1992). 

Institutional effectiveness requires "that we document whether or not our 
educational programs contribute to the attainment of important educational goals over 
and above the 'input' characteristics students bring to college" (Hanson, 1992). The 
range of assessment objects to determine levels of institutional effectiveness is as broad 
as the institution's goals themselves. Typical objects include student developmental 
change (quality and accessibility of counseling and advising services), faculty 
accomplishments (published scholarship, research and development achievements, 
public and community service effects), service to specialized constitutencies (targeted 
programs for legal assistance to the poor), and institutional climate (satisfaction levels, 
financial capacity, and the like). Student outcomes, of course, are a component of 
institutional effectiveness. To move toward a more inclusive academic achievement 
criterion in accreditation decisions, one regional organization declared that "evaluation 
of overall institutional effectiveness is dependent upon the institution's documentation 
of how well it is accomplishing not only its educational purposes, but also all other 
purposes and objectives needed in order to fulfill its mission" (Commission on 
Institutions of Higher Education, 1993). 

STUDENT OUTCOMES: Assessment of student achievement is not something new. On 
the contrary, it was common in 19th century colleges to require a fourth year oral 
examination as a method of validating the learning of the graduates (Marchese, 1987). 
In post- WWII American higher education, validation through performance gave way to 
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credit-hour accumulation as a proxy measure for competency attainment in 
undergraduate areas in other than the professional programs. 
Student outcomes could be broadly defined as: 

The wide range of phenomena that can be influenced by the educational 
experience (Jacobi, Astin, and Ayala, 1987, p. 19). 

This unwieldly definition may be refined as follows: 

The specific set of highly valued characteristics which the university, 
school or academic and co-curricular support services claim to instill by 
means of their programs (Nedwek, 1992). 

For example, if an institution seeks to prepare graduates with a demonstrated skill in 
written and oral communication (a goal statement), then an expected student outcome 
might be that "prior to graduation, each student will demonstrate expository writing 
with correct grammar, punctuation, and logical organi2ation....and [will demonstrate] 
the ability to organize and deliver a clear and substantive presentation (Southern 
Association of Colleges and Schools, 1989, pp. 26, 28). 

It is left to the key stakeholders to respond to basic questions about expected 
educational results such as: 

1. What knowledge must be mastered? 

2. Which skills are highly valued and must be demonstrated, e.g., problem- 
solving, oral conmaunication, working with groups, or what? Equally 
important, at what levels of proficiency should these skills be demonstrated? 

3. Which student attitudes and values are to be nurtured? Should they become 
conditions of graduation? 

4. When should a given quality be present? Which skills are manifest after the 
core curricular experi^ ace, at graduation, one year out, five years or when? 
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In student outcomes we need to look not only at the "what" questions, but the 
"how" issues as well. For example, assessment of student performance has taken many 
forms. Inside or outside the classroom, assessment efforts have included standardized 
tests, robust simulations, "rising junior" exams, senior exams, alumni surveys, 
graduating senior interviews, portfolio analysis, self-assessment, and juries of faculties 
or outside professionals (Hutchings and Marchese, 1990) 

The NCA's recent statement on academic achievement assessment provides an 
instructive framework: 

The Commission on Institutions of Higher Education of the North 
Central Association of Colleges and Schools reaffirms its position that 
assessment is an important element in an institution's overall evaluation 
processes....Assessment is not an end in itself, but a means of gathering 
mformation that can be used in evaluating the institution's ability to 
accomplish its purposes in a number of areas. An assessment program, 
to be effective, should provide information that assists the institution in 
making useful decisions about the improvement of the institution and in 

developing plans for that improvement With this statement we make 

explicit the Commission's position that student achievement is a critical 
component in assessing overall institutional effectiveness (Commission 
on Institutions of Higher Education, 1991, p. 393). 



The typologies of student outcomes are as diverse as the measurements 

themselves. Some argue for a model based on skill developmental areas, such as 

cognitive skills, attitudes, and behaviors after college (Ewell, 1985). Another sets forth 

individual goals and societal goals (Bowen, 1978); or cognitive and affective outcomes 

(Astin, 1991). An ambitious application is Alverno College's competency-based liberal 

arts program that assesses eight outcomes ranging from personal effectiveness skills to 

citizenship training. Bogue and Saunders offer a crucial insight about the Alverno 

approach. They state (1992, p. 169): 

The strengths of the model are that it is specific to the educational goals 
of the institution, that it involves the allegiance of the faculty 
development and implementation, and that it is directly linked to the 
curriculum and intructional processes of the institution. 

13 

ERIC 



Performance Indicators Page 13 



Thus, Alverao College represents one of the few models that provides a clear "fit" 
between the curriculum and outcomes. Equally important, this model integrates 
institutional goals with process, outcomes, assessment, and faculty development. The 
result is a model that demonstrates quality in performance (Townsend, Newell, and 
Wiese, 1992). 

PROCESS: Although institutional effectiveness and its various components are core 
concepts in the quality/accountability movement, the least understood notion is the 
linkage between inputs and outcomes, that is, process variables. One of the major 
leaps of faith in the field of outcome assessment is the often untested assumption that 
the academic program, institution, or organizational culture was in fact responsible for, 
or explains variations in, student outcomes and performance. This leap of faith 
disregards the question of whether or not the intended learning actually occurred. 
Process assessment attempts to "get behind" outcomes to enhance the credibility of the 
claims of success (Hutchings, 1990). 

The literature reflects considerable definitional confusion about process. From 
a program evaluative perspective, process refers to the "treatment", or what actually 
happened during the program (Rossi and Freeman, 1989). When applied to 
educational programs, this evaluative orientation asks such questions as the degree of 
conformity with program design, the extent of coverage or exposure to the program, 
and among others, the timing and logic of the individual elements. Simply stated, the 
focus is on the "what" question, or what specific contribution did the institution and 
program make to the achievement of student outcomes? 

Process assessment is essential to document the degree of fidelity to the original 
design. It generates baseline information about what service or curriculum was 
provided to which students by faculty of varying levels of competence. The problem of 
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the "nonprogram" in the delivery of services has been well documented in the 
evaluative literature (Rossi, 1978), For example, the use of rising junior exams as a 
component of intermediate outcome assessment may have limited utility if a large 
proportion of the freshmen or sophomores are transfers from other institutions. The 
increasingly conraion pattern of mobile freshmen and sophomores suggests that a 
lessening proportion would have experienced the same core curriculum. Under the 
condition that the treatment (i,e,, core curriculum) was not experienced by the vast 
majority of a group of upper classmen, how can one attribute student 
outcomes/achievements to the program? In reality, the program never happened, 

A recent discussion about quality in higher education makes the point about 
process even more convincingly* As one student of Deming remarked (Dill, 1992, p, 
43): 

Deming emphasizes that the improvement of quality does not come from 
inspection, or what in education might be termed assessment, but from 
design-from the continuous improvement of the underlying processes of 
production. 

Deming*s point is critical to understanding the need for process assessment that 
incorporates design review. A focus on design is essential to understanding TQM's 
view of quality by reducing variation; in this case, variation in program delivery. Thus, 
Deming^s notion of "common causes" as sources of variation in quality provides a rich 
base for building process performance indicators, such as poor design, poor materials, 
inefficient technologies, ineffective supervision, and the like (Dill, 1992), 

In many situations, assessment lacks coherence because of the degree of 
measurement fragmentation of programs, policy and process (Banta, 1993), In Europe, 
for example, universities in the OECD countries appear uninterested, if not unable, to 
develop interests of faculty beyond their disciplinary home (OECD, 1987), The 
pattern in North America, especially the United States, is equally uneven. The 
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individualism and autono^ay of the academic culture in the American tradition 
undermines the homogeneity of the curriculum and, in turn, the ability to account for 
the variance in outcomes. Paradoxically, there exists a natural tension between the 
drive toward coherence of process and faculty autonomy. However, recent work in the 
development of performance indicators in Scotland provides an example of process 
assessment measurement. A Scottish initiative identified eleven distinct dimensions 
for the periodic assessment of the learning environment (Scottish Higher Education 
Funding Council, 1992). This approach might lend itself to sampling techniques of 
curricula, syllabi, and the like that can be "scored" into a programmatic measure (the 
percent of programs within an institution that meet minimum standards). 

An alternative perspective on process involves a look at the provision of 
services. Teaching evaluations and client satisfaction measurement are popular 
expressions of this orientation. This perspective promotes the "market" orientation of 
organizational control. Although client satisfaction measurement is an attractive 
approach to process assessment, research suggests that satisfaction is a function of the 
congruence between expectations and experience. Process measures, however, seldom 
focus on both expectations of service and the actual experience of it (Steers and Porter, 
1983; Neal, 1990). 

A third perspective involves the introduction of conversion ratios as proxy 
performance indicators. Ratio analyses have become very popular in assessing an 
institution's financial condition. Their growing popularity is not surprising, given the 
proportion of corporate executives that are members of college and university boards 
of trustees. They have a working familiarity with these tools. For example, the ratio of 
expendable fund balances to total expenditures and mandatory transfers is used to 
gauge an institution's strength relative to its operating size (KPMG Peat Marwick, 
1990). The use of process or operating performance indicators received considerable 
impetus from early work in Great Britain. Indicators included staff/student ratios, 
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class sizes and the like (Jarratt Report, 1985). Dutch polytechnics have been assessed 
using a variety of cost measures, such as average expenditure allocated to instruction, 
support staff, student services, and the like (De Jager, 1992). Non-standardized 
recording techniques, however, undermine the comparability of financial ratio analyses 
(DiSalvo, 1989). 

Process measures are far less likely to generate consensus as performance 
indicators than "input" or resource indicators. University management statistics in 
Scotland, for example, include a wide array of resource indicators, but little in the way 
of teaching performance (Gordon, 1991). When process measures are part of a 
discussion of faculty productivity, the concept of faculty time is highly influenced by the 
academic culture, especially group norms and "perceived property rights" (Massey and 
Wilger, 1992). Simply stated, time-on-task measures (contact hours, advising and 
mentoring) are strongly influenced by shared beliefs about the faculty's perception of 
their environment. 

The basic concepts investigated in this section provide key issues to consider 
when developing perfonnance indicators. Consequently, these concepts also assist in 
the development of criteria for the assessment of indicator systems. The key issues 
raised in this review of basic concepts include: 

Quality: institutional mission; goal achievement 

Assessment: measurement; expected results 

Indicators: quantitative statistics; comparative ranking; institutional 

functions; conceptual framework 
Effectiveness: educational goals; mission objectives 
Outcomes: valued characteristics; demonstrated skills; methodology 
Process: treatment; contribution; design; variation 
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These issues provide an extensive foundation for the construction of criteria for 
indicator assessment. The following section discusses the dimensions selected for 
comparison. 

COMPARATIVE CRITERIA 

To assist in the assessment of performance indicator initiatives in Europe and North 
America, nine dimensions were selected as comparative criteria. These dimensions 
include: 

1. Lx)cus of Control 

2. Degree of Governmental Involvement 

3. Focus of Performance indicators 

4. Sources of Quality Variation 

5. Data Selectiori 

6. Intended Audiences 

7. Emphasis of Use 

8. Impact on Student Learning 

9. Relationship to Institutional Mission 

Each dimension represents a critical concern raised in the research literature on 
performance assessment or quality management. The combined framework will assist 
in understanding the characteristics of individual initiatives, while permitting 
comparative analysis between initiatives. 

Locus of Control ! De Jager (1992) asserts that higher education's dual system 
of governance (administrators and professions) limits the usefulness of traditional 
management techniques in many higher education settings, that is, the strategic 
"control" mechanisms. Ouchi (1980) classifies these mechanisms into three categories: 
clans, bureaucracies, and markets. Clans seek to control organizational behavior 
through shared values, traditions, and social cultures, such as those found among 
faculty in departments, schools, or universities. Bureaucracies control through 
hierarchical authority and rules, such as state boards or accrediting agencies. Markets 
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rely on forces, such as competition and pricing, to shape organizational behavior. Dill 
(1992) characterizes the British system of higher education as largely a clan model of 
control, while other European systems rely on x more bureaucratic approach. The 
American system has evolved from a clan form of control, through bureaucratic 
approaches, and now exhibits a market-driven mechanism. Part of the issue of locus of 
control is the role of faculty in establishing assessment criteria and methodology. 
Faculty organizations in the United States are beginning to express their concerns over 
standards for mandated assessments at the state level (Council of the American 
Association of University Professors, 1991). 

Degree of Governmental Involvement : The role of government in the affairs of 
the academy is shaped by a variety of environmental factors. Two of the more 
powerful forces are the relative scarcity of resources and the "political culture" of the 
state and its resultant governance pattern (Ewell and Boyer, 1988). The degree of 
governmental control over performance indicator systems and other assessment 
schemes is colored by policy makers perceptions of "the problem" and "the solution." 
These perceptions, in turn, shape the form and substance of assessment systems. For 
example, in the state of Missouri, the initial impetus for assessment was portrayed by a 
ideological governor as an element of the reform of higher education. The relatively 
centralized and active centralized decision making in states such as South Dakota and 
New Jersey stand in sharp contrast to efforts underway in Virginia, or the balkanized 
actions in Colorado (Ewell and Boyer, 1988). 

Government involvement can range from direct to indirect to a laissez-faire 
position. The most penetrating or direct degree of involvement would be state 
mandated assessment systems with prescribed methodologies for performance 
indicator design and use. Indirect control is characterized by a degree of delegation to 
postsecondary institutions that allow for implementation diversity within the context of 
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broad goal statements. A laissez-faire position descibes the con iition where the state 
or national leadership is either disinterested in the educational policy arena, or where 
coordinated efforts are largely voluntary in character. While European and American 
State systems of higher education share the characteristic of direct governmental 
control and funding, even the American "independent" sector faces indirect 
governmental involvement in the assessment of institutional performance. Through 
the reauthorization of the Higher Education Act (1992), the U.S. Congress has 
introduced new opportunities for state review of a wide variety of institutional 
measures. Some private institutions, however, may avoid this review and will continue 
a more laissez-faire relationship with government agencies. 

Focus of Performance Indicators : Prior to the selection of data to create 
performance indicators, the researcher must determine the underlying focus of the 
planned analysis. A number of observers (Dochy and Segers, 1990; Bell and 
Wilkinson, 1992) assert that performance indicators must be related to the functions 
and goals of the institution. In relationship to institutional goals, the focus of 
performance indicators may be on documenting efTectiveness (how well the institution 
accomplishes its goals), increasing efHciency (what it costs to achieve those goals), or 
improving economy (how to save money, yet achieve the goals). The process of 
developing indicator systems should begin with a the development and application of a 
conceptual framework that is "based on research results and the interests of 
policymakers and educators" (Blank, 1993). 

Sources of Quality Variation : Deming*s (1986) philosophy of continuous and 
systematic improvement of organizational production and service stresses the 
importance of reducing variation in production design and processes. Dill (1992) 
contrasts institutional assessment of special causes of variation, which are attributable 
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to individuals, with common causes, which are attributable to poor institutional 
processes, such as design, materials, technologies, and supervision. Dill contends that 
assessment of special causes will have little effect on organizational performance, but 
assessment of common causes, or processes, will have a substantial impact on the 
quality of the institution. Chaffee and Sher* (1992) characterize special causes as 
exceptions to the normal process that require quick detection and administrative action 
to eliminate. Common causes are inherent in every process and are not attributable to 
the worker. To eliminate common causes, Chaffee and Sherr call for a data-driven, 
scientific approach that involves participants and crosses organizational boundaries. 

Data Selection : Recognizing the need for comprehensive data to support 
performance assessment, a number of authors (Ball and Halwachi, 1987; Scottish 
Colleges, 1992; Chaffee and Sherr, 1992) call for the selection of input, process, and 
outcomes data elements. Johnes and Taylor (1990) suggest that most institutions 
concentrate primarily on input measures when assessing quality. The student 
outcomes assessment movement in the U.S. has increased American awareness of 
outcomes data as well, but has created few process measures. Chaffee and Sherr 
(1992) emphasize the need to develop process assessment to identify common causes 
of variation. While numerous performance indicator ventures assess input and output 
phenomena, little is done with process variables. Often the problem with process 
measures is the lack of valid and reliable data, coupled with the cultural norm that 
faculty are not required to publicize their methodology of assessment in various 
learning situations. 

Intended Audiences : Higher education institutions produce performance 
indicators for specific audiences (board, government, faculty, students). The following 
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typology categorizes these audiences by relationship to the institution and anticipated 
outcome of the information: 

Internal Decision-Makers (Administration/Faculty) 

Internal Policy Makers (Board) 

External Decision-Makers (Parents/Students) 

External Policy Makers (Government Agencies/ Accreditation Bodies) 

Emphasis of Use : The quality improvement literature (Deming, 1986; Dill, 
1992; Ewell, 1992a) suggests that performance indicator systems must move from 
certifying competence to improving institutional quality. As discussed above, Darling- 
Hammond (1992) offers four primary purposes for performance indicators, e.g., (1) 
Monitoring general conditions and contexts; (2) Identifying progress toward specified 
goals; (3) Illuminating or foreshadowing problems; and (4) Diagnosing potential 
sources of identifled problems. In addition to these four purposes, two additional 
purposes can be served. A fifth use is tied to institutional funding, namely, incentive 
funding and resource allocation. In Great Britain, for example, the relationship 
between indicators and resource allocation decision making was expressed in the mid- 
198as (Green Paper, 1985). 

A sixth use centers on political gains. The policy analysis literature has 
documented political use as one of the major applications of information (Ewell, 1989; 
Kalsbeek, 1991; Leviton and Hughes, 1981; Nedwek and Neal, 1992; Weiss, 1977). 
Performance indicator systems can serve the political agenda of decision makers who 
are concerned that they are "doing something" about postsecondary education. The 
danger is the degree of penetration or intrusion by the state into the affairs of 
postsecondary education (Newman, 1987). 

In summary, the types of use under this criterion include: (a) monitoring 
conditions, (b) measuring progress, (c) forecasting problems, (d) diagnosing problems, 
(e) allocation decision making, and (f) political symbolism. 
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Impact on Student Learning : Perhaps the most frequently overlooked 
dimension in the comparative analyses of performance indicator systems is the effect of 
a model on student learning. This dimension is offered because of the need to remain 
sensitive to the primary objective of the industry, namely, facilitating student learning. 

Quality assurance models imply that a system can have direct, indirect, or 
assumed effects on student learning. Direct impact exists where the model 
intentionally yields data that validates or provides evidence of student achievement. 
Indirect impact represents models that promote a "culture of evidence" that fosters a 
concern about variations in outcomes and linkages back to program delivery. Assumed 
impact represents the most common model; one where student outcomes represent a 
leap of faith from the program outline. Perhaps the most familiar leap of faith is a 
model where productivity improvements somehow yield enhancements to the quality of 
the academic enterprise (Massey and Wilger, 1992). 

Relationship to Mission : Performance indicator models vary in the degree to 
which they are related to programmatic, institutional or system-wide missions. The 
relationship may be direct, indirect, or assumed. A direct relationship exists where the 
mission statement of the institution promotes the distinctiveness of the enterprise. 
Thus, the statement should include clear goals about intended outcomes, clientele to 
be served, and steps insure achievement of the mission (Newsom and Hayes, 1990- 
91; Bogue and Saunders, 1992). An indirect relationship exists where the mission 
statement and performance assessment system suggest standards, but with less 
precision. Assumed relationships occur where mission articulation is independent of 
performance indicator models. 
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DATA ANALYSIS 

Following a review of institutional reports, government documents, and the 
research literature on evaluation and assessment, a sample of approaches to generating 
performance indicators was examined using the nine comparative criteria. The six 
systems in the sample were chosen as representative of various segments of higher 
education. 

National Education Goals: U.S. National System 
Critical Choices: U.S. Statewide System 
Key Success Indices: U.S. Institutional System 
CVCP: European National System 
Queen^s University: Canadian Model 
MONEY Magazine: U.S. Media System 

Table One summarizes the findings of the analysis. In addition, the narrative 
provides a separate assessment of each system, with an evaluation of strengths and 
weaknesses inherent to each approach. The analysis concludes with a discussion of 
similarities and differences between the systems, highlighting major implications for 
researchers and administrators. 
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Great Britain's CVCP: Since 1987, the Committee of Vice-Chancellors and Principals 
(CVCP) has published an annual listing of University Management and Performance 
Indicators for all universities in Great Britain. Originally designed by chief academic 
administrators to strengthen the public image of the university system, this listing has 
attracted attention from government officials and funding bodies for monitoring and 
resource allocation. The performance measures are predominantly input variables; 
two indicators are outcome variables (graduation rates and destinations of graduates). 
Cave, Hanney, and Kogan (1991) consider the system more useful in assessing 
efficiency than effectiveness. The system assumes an impact on student learning 
through adequate resources and measures increased performance through higher 
graduation rates. 

Critical Choices: This 24 goal initiative is the work of a state coordinating board with 
the strong support of the former governor. The majority of the goals and performance 
indicators are either prescriptions or admonitions about resources and resource 
allocation. Government involvement is especially direct through restrictions on 
admissions policies. Data are developed at the institutional level for comparative use 
by internal and external decision makers at the system level, especially among public 
institutions. Although the relationship between performance and incentive funding is 
mentioned (Goal 20), the role of the coordinating board in the funding process may be 
ineffective and somewhat symbolic. The impact on student learning is assumed to 
result from changes in resource allocation decisions. Finally, in an unusual 
interpretation of the role of institutional missions, the model encourages mission 
statements to represent admissions standards rather than progranmiatic initiatives. 
Inter-institutional comparisons are only beginning to be implemented. The 
Association of Universities and Colleges of Canada (AUCC, 1993) may become a 
major player in developing indicators and the rationale for comparative reporting. 
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University of Miami's Key Success Indices: This senior management tool represents 
one of the more detailed efforts at monitoring performance across the range of 
institutional functions. Primary focus is improvements to the effectiveness and 
efficiency of the university. Data are collected at the systemic level for use by internal 
policy makers. Use seems to emphasize monitoring existing conditions against prior 
year YTD data and some concern about forecasting future trends. Although the model 
deals with a rich variety of indicators, the connection to student learning and 
institutional mission is assumed. 

National Education Goals: The U.S. Department of Education published a list of 
National Education Goals in 1992. Goal 5 addresses the need for all Americans to 
possess the knowledge and skills necessary to compete in a global economy. Two 
objectives under Goal 5 call for the creation of institutional performance indicators: 

(a) student retention and completion rates, especially among minority students; and 

(b) demonstrated outcomes in critical thinking, communication, and problem solving 
skills. Institutional data will be combined into state reports for comparative analysis. 
While still under development, the plan calls for the development of an assessment 
system to measure the effectiveness of postsecondary education. Tliis plan faces a 
number of challenges, such as: 

1. How to quantify "world-class standards." 

2. Developing valid measures of skill development. 

3. Differentiating between goal achievement and skills development 
(product vs. process assessment). 

4. Maintaining diversity of mission while imposing national "standards." 
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Queen's University: This model emerges from a twenty year history of higher 
education data capture and dissemination practices in Canada, Organizational control 
is hierarchical with relatively indirect influence of provincial govenurtent. Indicators 
emphasize effectiveness measurement, although a few appear related to public 
relations and diversity goals (gender balance, geographic diversity). Vast majority of 
measurements examine resources (qualities of students and reputations of faculty). 
Some measures may be proxies for process variables (expenditures per PTE student 
for student services, libraries, acquisitions, central computing). Intended audiences 
flow from a marketing orientation of the indicators (students selection of an 
institution), and from a monitoring perspective (tracking resources). Relationship to 
student learning measurement is assumed. The relationship to institutional mission is 
somewhat indirect. The connection is more direct for indicators of research 
productivity. 

Money Magazine College Guide: This guide represents the market driven approach to 
performance indicator tools. Although a host of substantial methodological problems 
are associated with this model (Webster, 1992), the key characteristic is the tenuous 
correlation between the perception of quality and "hard data on product performance" 
(Bogue and Saunders, 1992, p. 66). The differences between these two sources of data 
about quality help us understand the current political debate on rankings. Typical of 
market sensitive tools, this model emphasizes institutional resources (percent of 
freshmen with high test scores, student/faculty ratio, entrance exam results, student 
services budget). Data to support rankings of the "best buy for the dollar" are taken 
largely from public reports. Given the market orientation, their use is largely symbolic. 
Similarly, a conceptual framework, defense, or explanation for the selection of schools 
and indicators assumes an impact on student learning. 
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DISCUSSION: 

This sample of performance indicator initiatives yields several common themes. 
First, performance indicators, for the most part, are variations of input/output 
mechanistic thinking. This orientation may be explained by the accountability theme 
as played by budgetary processes. Appropriations decision-makers are accustomed to 
trade-offs over resources because they are easily "priced" in the policy marketplace. 
Process indicators are not normally expressed in similar terms. These indicators most 
closely lend themselves to allocation efforts as proxy measures of process. Typical 
proxies that are convertible include class-size and full-time faculty per PTE student. 

Second, a remarkable similarity of effort exists regardless of settings in North 
America and Europe. Despite enormous variations in the contexts of higher 
education, the overall design and implementation strategies are quite similar. In 
media systems, for example, the "league tables" are nearly identical in format, 
substance, dissemination, and validity. Applications in Great Britain, Canada, and the 
United States are filling the accountability void with highly successful marketing 
tabloids. The systemic approaches to performance indicators in Europe and North 
America share numerous characteristics, despite differences in size, scale, and 
complexity of state or national models. 

The use of comparative criteria generated a number of observations. First, 
governance and organizational control patterns appear to be more hierarchical at the 
outset of model development, but appear headed toward a market orientation. 
Second, performance indicators emphasize resources and gross outcomes for use by 
policy makers and decision makers in monitoring existing systems. Although progress 
toward goal attainment (a focus on effectiveness) is alluded to in most models, the 
absence of measurable goals and objectives limits the power of this focus. 

The emphasis of use in these models is more often to monitor or review current 
operations, oftentimes attempting comparisons despite the absence of consistency in 
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definitions or data capture practices. Such an emphasis is consistent with the 
observation that most models are unprepared to address conversion or process 
variables. Third, these models are built on a "leap of faith" that outcomes or 
performance levels observed can be attributed to something in the system, institution, 
or learning environment. The current architecture of these models does not support a 
developmental perspective. On the contrary, the absence of linkages back to the 
learning environments leaves internal decision-makers without information to "correct" 
or modify the common causes that explain variations in quality. Thus, the probability 
of developing actionable policy or programmatic recommendations based on 
information yielded from these systems is highly unlikely. 

Fourth, because an impact on student learning is assumed to occur as a result of 
introducing any of these models, external policy makers will be denied specific 
information for either formative or summative evaluation. The danger is that future 
development of performance indicators will be more symbolic than functional. 



STUDY IMPLICATIONS: 

Institutional effectiveness is the broad umbrella for measuring a postsecondary 
organization's achievement of expected educational results through performance 
indicators. Institutional, program, and student outcomes form the base of institutional 
effectiveness, and ideally, flow from mission and goal statements. 

This study raises several danger flags about assessment and performance 
indicator efforts. First, researchers, system designers, and decision makers must guard 
against easily quantifiable and readily available methods; a sort of one-size-fits-all 
approach. Decisions must be made about which outcomes are more important than 
others, that is, distinguishing the essential from the desirable. Institutional researchers 
and policy makers must be aware of the meanings that students, faculty, alumni, 
employers and others bring to performance indicator and outcomes assessment 
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initiatives. Equally significant, system designers must be receptive to the wide range of 
forces inside and outside the classroom across time that help shape the student: 

Assessment...looks less like a graduation snapshot than it does like a 
movie: "scenes" from the student's college experience, behavior and 
achievement over time, seen in multiple settings and contexts, from 
various points of view, sorted out and interpreted oy thoughtful, involved 
audiences-including students themselves (Hutchings, 1989, p. 4). 

Second, while this brief review and analysis suggests an inherent tension 
between quality improvement and external accountability mandates from accreditation 
commissions or governmental agencies, the undisciplined acceptance of "off-the-shelf 
assessment instruments in support of performance indicator models and processes is 
short-sighted: 

Assessment per se guarantees nothing by way of improvement, no more 
than a thermometer cures a fever. Only when used in combination with 
good instruction (that evokes involvement, in coherent curricula, etc.), in 
a program of improvement, can the device strengthen education 
(Marchese, 1987, p. 8). 

A third problem with performance indicator systems is the absence of a strong 
conceptual framework that supports tracing outcomes back to the learning 
environment. Effective models of higher order skill development require a strong 
theoretical base. One description of a national assessment effort suggests: 

[T]he most effective assessment of learning takes place when a theory or 
model of how instruction will lead to critical thinking or problem solving 
has been defined and tested (U.S. Department of Education, 1992, p. 6). 

Faculty must be invited to lead in developing performance indicator systems, just as 
policy makers and decision makers need to articulate the intended uses of performance 
indicator data. Although there are several positive uses of performance indicators, 
such as building a common language for formative evaluation, (Nadeau, 1992), the 
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academy may quickly perceive such initiatives as punitive tools of an insensitive 
bureaucracy. 

Fourth, care must be given to building the standards or peformance 
benchmarks. Standard-setting is a normative exercise with substantial consequences 
for key stakeholders in higher education. Perceived arbitrariness in standard-setting 
can undermine the credibility of the effort. The goals and objectives of the "Critical 
Choices" program in the State of Missouri illustrates the point. One of the goals 
addresses the development of national recognition of the state's universities by 
achieveing several objectives. One objective calls for: 

[I]ncreasing by 50 percent, by 1996, the amount of money awarded on a 
competitive basis to Missouri's public research universities from both the 
federal government and other external sources for basic and applied 
research grants and contracts (McClain, 1992, p. 10). 

Typical of most indicators in this model, the rationale behind setting the increase at 50 

percent was never discussed nor disseminated to faculty or academic administrators. 

As with so many models that rely exclusively upon quantitative indicators, the danger 

exists that decision makers may begin to believe that numerical statements are 

objective expressions of existing or desired conditions. 

Fifth, because the ease of "roll-up" through levels of aggregation affects system 
design alternatives, information building requirements of performance indicator 
systems must include a range of standard activities from selection, capture, 
manipulation and reporting (Howard, McLaughlin, McLaughlin, 1989). Systems 
lacking the capacity to report through levels of aggregation have far more limited use 
than those with integrated databases. 

Sixth, indicators may not necessarily act unidirectionally nor maintain positive 
interrelationships. For example, it is important to recognize the probable inverse 
relationship between rankings based on faculty publications and the quality of 
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teaching; that is, reputations associated with faculty publication, and those rankings 
based on some regional or national distinction in curricular matters or learning 
climate. It seems paradoxical that highly innovative institutions require an 
extraordinary commitment from the faculty to install and sustain a substantial 
pedagogical innovation. The little research that is produced by faculty in these 
situations is confined to formative evaluations of some type. Such scholarship and 
involvement may be unrelated to rank/tenure and promotion decisions. Although such 
work is of little use to the discipline-based outlets of traditional scholarship, a 
recommendation has been made to increase the status of such effort (Boyer, 1990). 

This study has implications for the profession of institutional research as well. 
Offices of institutional research are under increased pressure to capture and report 
timely information on institutional effectiveness. Several factors can increase the 
capacity f the institutional research community to respond in a timely manner to the 
plethora of requests for performance data. First, institutional researchers have a 
fiduciary responsibility to help develop the conceptual underpinnings of the 
performance indicator movement. Their role must be expanded to that of an 
institution. ^*«.wiiropologist (Nedwek, 1993). Senior researchers have the opportunity 
to explOi ; : ' npeting meanings of the core terms of higher education (academic 
success, quality), and have an obligation to bring alternative conceptual perspectives 
before decision makers and policy makers inside and outside the institution. 
Institutional researchers need to embrace the anthropological role in information 
building. The challenge for researchers is a "matter of defining one's role not as a 
chronicler but as a lively facilitator and interpreter of the communil process of self- 
discovery and self evaluation" (Gould, 1992, p. 49). 

Although the higher education literature provides analytic distinctions among 
research action, reconmiending policy, and monitoring environments (Gill and 
Saunders, 1992), building effective performance indicator systems force an integration 
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of the traditional roles assumed by institutional researchers and policy analysts. Staff 
are cautioned to move beyond the "inward looking nature" of the profession by 
providing data in support of fund-raising, public and government relations (Jones, 
1991). 

Second, institutional researchers must be prepared to deal with varying levels of 
understanding and readiness among academic stakeholders to support performance 
indicator processes. This study has shown widespread problems of measurement 
(psychometric properties, internal validity, timing of program effect). Similarly, most 
initiatives make the leap of faith between resource indicators and performance 
outcomes. The conceptual difficulties may create a need for extensive retraining of 
staff that will enable them to address these complexities. 

Third, researchers must be prepared to deal with the inherent contradiction 
between a steady, slow process of internal review of the academy with the demands for 
quick results (Ewell, 1987). A recent reaccreditation self-study remarks: 

The state agency demands for 'evidence* of institutional effectiveness, 
coupled with the expectations of accreditation bodies, create 
considerable pressure on the offices of institutional research to embrace 
outcome measures that are based more on data availability than on the 
authenticity of the assessment (Nedwek, 1993, p. 39). 

IMPLICATIONS FOR FUTURE RESEARCH: 

As a work in progress, this study sought to describe the current state of affairs in 
the design and implementation of a variety of performance indicator systems in Europe 
and North America. Future efforts will expand the number and type of cases and apply 
the nine comparative criteria developed in this paper. Of special interest will be 
developments in Finland, the Netherlands, and selected middle-European countries. 
North American cases will be expanded to include more mature initiatives, such as the 
Tennessee model. 
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